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1.  INTRODUCTION 


The  geo-political  changes  around  the  world  in  the  last  five  years  have  clearly  shown  the  need  for 
the  U.S.  to  maintain  a  strong  technical  edge  in  high-tech  weaponry  over  potential  adversaries. 
Unlike  the  thrust  on  ballistic  missiles  technology  seen  earlier,  before  the  dissolution  of  Soviet 
Union,  the  recent  thrust  is  more  on  accurate  targeting.  Targeting,  particularly  in  the  case  of 
relocatable  targets,  is  expected  to  pose  a  considerable  challenge  to  future  missions  whether  manned 
or  unmanned.  Relocatable  targets  such  as  mobile  launchers,  armored  vehicles,  ammunition  and 
equipment  carriers,  supply  trucks,  and  mobile  radar  units  played  a  crucial  role  in  the  Gulf  war  by 
providing  a  difficult-to-eliminate  capability  of  the  adversary.  Use  of  smart  weapons  in  that  war 
provided  a  substantial  advantage  to  the  Allied  Forces.  Although,  this  experience  demonstrated  the 
vast  potential  of  high-tech  weaponry  and  smart  sensors,  it  also  helped  point  out  the  deficiencies  in 
current  systems,  in  particular,  the  need  for  robust  and  reliable  Automatic  Target  Recognition  (ATR) 
technology  to  improve  the  strike  accuracy. 

The  goal  of  robust  ATR  can  be  achieved  if  we  can  make  use  of  multiple  sensors  in  a 
complementary  fashion.  While  sensors  such  as  Synthetic  Aperture  Radar  (S  AR)  which  provide  a 
wide  area  of  coverage  obviously  suffer  from  poor  resolution,  sensors  such  as  Forward  Looking 
Infrared  (FLIR)  which  provide  a  good  resolution  suffer  from  a  narrow  field  of  view.  In  addition 
to  this,  to  facilitate  the  all-night-all-weather  targeting  capability,  we  need  to  make  certain  that 
optimal  use  is  made  of  the  strengths  of  different  sensors.  In  this  sense,  use  of  SAR  for  detection 
of  targets  has  become  very  popular.  Since  SAR  suffers  from  poor  resolution  and  lack  of  look- 
angle-independent  features,  we  need  to  complement  the  SAR  with  FLIR  or  electro-optical  (EO) 
sensors  for  accurate  target  recognition. 


For  over  a  decade,  a  number  of  ATR  initiatives  such  as  the  Reconnaissance,  Surveillance,  and 
Target  Acquisition  (RSTA),  and  Moving  and  Stationary  Target  Acquisition  and  Recognition 
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(MSTAR)  program  funded  by  the  Defense  Advance  Research  Projects  Agency  (DARPA),  have 
resulted  in  very  mature  image  understanding  and  sensor  fusion  algorithms  at  academic  institutions 
such  as  the  University  of  Maryland,  College  Park,  MD.  This  STTR  Phase  I  is  essentially  a 
technology-transfer  initiative  to  incorporate  mature  detection  and  multi-sensor  registration 
techniques  developed  at  the  University  of  Maryland  into  a  robust  model-based  ATR  module 
developed  at  LNK. 

This  report  is  organized  into  eight  sections.  In  section  2,  we  discuss  the  Phase  I  technical 
objectives  and  summarize  our  work  in  reaching  these  objectives.  In  Section  3,  a  conceptual 
framework  for  a  SAR-FLIR  multi-sensor  fusion  prototype  system  is  discussed.  A  computationally 
elegant  registration  scheme  is  presented  in  Section  4.  Section  5  presents  a  discussion  on  the 
Constant  False  Alarm  Rate  detection  algorithms  used  in  the  Phase  I  effort.  The  model-based  vision 
technique  is  described  in  Section  6.  We  discuss  the  results  in  Section  7  and  lay  the  foundation  for 
the  Phase  n  work  in  Section  8. 


2.  A  SUMMARY  OF  THE  PHASE  I  WORK 

Synthetic  Aperture  Radar  (SAR)  platforms  have  been  found  very  useful  for  target  detection 
applications  due  to  their  all-weather,  and  all-day-and-night  capabilities.  However,  unlike  Electro- 
Optical  (EO)  sensors,  SAR  imaging  systems  are  limited  by  their  lack  of  preserving  irradiance 
characteristics  of  the  terrain  objects.  Serious  difficulties  arise  when  we  use  SAR  for  recognition  of 
objects  because  it  is  difficult  to  extract  good  geometric  features  from  SAR  imagery.  Passive 
sensors,  on  the  other  hand,  remove  this  difficulty  at  the  cost  of  a  very  narrow  field  of  view. 
Therefore,  it  makes  all  the  more  sense  to  use  SAR  to  locate  potential  areas  where  targets  are  likely 
to  be  present  or  simply  Focus  of  Attention  (FOA),  and  direct  the  passive  sensors  to  obtain  high 
resolution  target-specific  details  from  the  areas  of  FOA. 
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Our  primary  goal  in  this  Phase  I  was  to  assemble  a  suite  of  sensor  fusion  techniques  for  SAR- 
passive  imagery  fusion  based  on  the  on-going  ATR  work  at  LNK  and  the  University  of  Maryland. 
In  reaching  this  goal,  our  major  technical  objectives  were  to: 

(1)  understand  the  platform  characteristics  (physics  and  geometry  of  imaging)  of  the  sensors 
involved; 

(2)  develop  the  techniques  for  extracting  FOA  from  the  SAR  imagery 

(3)  develop  registration  algorithms  for  SAR-FLIR  mapping; 

(4)  implement  the  model-alignment  algorithm  for  target  recognition; 

(5)  demonstrate  the  proof-of-concept  on  SAR-FLIR  data  available  inhouse;  and 

(6)  evaluate  the  ATR  performance  characteristics. 

We  have  devoted  a  lot  of  attention  to  meeting  each  of  these  objectives  in  this  Phase  1.  Our  concept 
of  using  the  SAR  as  the  cueing  sensor  for  locating  targets  and  recognizing  the  targets  is  shown  in 
Figure  2.1.  We  have  achieved  an  effective  technology-transfer  of  matured  algorithms  for 
detection,  multi-sensor  registration  and  recognition.  However,  due  to  the  lack  of  enough  data,  the 
sixth  objective  of  evaluating  performance  characteristics  needs  additional  attention  in  Phase  n  when 
we  acquire  more  data  sets.  In  essence,  the  following  are  the  major  contributions  resulting  from 
this  Phase  I  effort. 

•  A  Constant  False  Alarm  Rate  (CFAR)  has  been  developed  and  implemented  for  real-time 
performance  on  an  inexpensive  Single  Instruction  Multiple  Data  (SIMD)  hardware. 

•  An  innovative  registration  algorithm  has  been  developed  to  register  the  S  AR-passive  image 
pair. 
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Figure  2.1:  SAR  based  cueing  of  FLIR  for  detection  and  recognition  of 
ground  targets 
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•  A  model-based  recognition  scheme  using  a  model-image  alignment  approach  has  been 
implemented. 

•  An  end-to-end  demonstration  of  the  proof-of-concept  demonstration  of  the  SAR-passive 
image  fusion  approach  has  been  accomplished. 

We  show  the  results  of  each  of  the  three  major  steps,  namely,  detection,  multi-sensor  registration, 
and  recognition,  in  Sections  4,  5,  and  6.  Our  goal  in  Phase  11  is  to  develop  a  real-time  prototype 
ATR  system  that  exploits  complementary  strengths  of  SAR  and  passive  imaging  systems.  The 
framework  is  a  generic  one  in  that  we  combine  SAR  with  any  type  of  passive  sensor.  In  addition, 
if  the  resolution  of  the  SAR  is  sufficiently  high  we  will  exploit  the  SAR  data  also  for  ATR  in 
addition  to  using  the  SAR  for  detection  and  cueing.  A  more  sophisticated  SAR-passive  image 
registration  algorithm  which  takes  into  account  imaging  geometries  of  the  active  and  passive 
sensors  will  also  be  the  focus  of  the  Phase  II  work.  For  improving  the  ATR  performance  of  the 
Phase  I  approach,  we  will  work  on  an  indexing  scheme  for  model  selection  based  on  geometric 
invariants. 


3.  BACKGROUND 

Over  the  last  few  years,  several  approaches  to  sensor  fusion  have  gained  a  strong  momentum 
including  Bayesian  reasoning,  Dempster-Shafer  theory,  fuzzy  logic,  blackboard  frameworks, 
neural  networks,  and  hybrid  systems.  Though  our  objective  in  writing  this  section  is  not  to  cover 
the  entire  sf)ectrum  of  issues,  we  will  attempt  to  provide  a  short  summary  of  the  earlier  work  in  this 
section.  A  number  of  survey  articles  have  been  written  in  the  recent  past  (e.g.,  Luo  and  Kay 
[1992]). 
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Multi-sensor  fusion  can  be  addressed  at  least  at  four  different  levels,  namely.  Level  I,  Level  II, 
Level  III,  and  Level  IV.  Consistent  with  the  terminology  adopted  by  the  DARPA  JDL  sensor 
fusion  group,  the  following  connotation  applies  to  these  different  levels  of  fusion.  Level  I  fusion 
refers  to  fusion  of  information  from  similar  sensor  platforms.  Level  II  fusion  implies  the 
integration  of  information  from  dissimilar  sensors  spatially  and  temporally  separated.  Level  HI 
fusion  involves  inference  of  individual  activities  of  different  entities  in  the  theater.  And  Level  IV 
fusion  refers  to  the  process  of  integrating  and  inferencing  activities  of  multiple  individual  entities  at 
the  theater  level.  Our  work  in  this  context  is  a  Level  II  fusion  effort  and  therefore  we  will  only 
discuss  those  efforts  which  involve  Level  11  fusion. 

One  of  the  significant  advantages  of  the  multi-sensor  based  approach  is  the  complementary 
information  available  in  the  different  types  of  imagery  to  achieve  segmentation  of  objects  from  the 
background.  Camouflage  that  prevents  segmentation  in  visual  imagery  may  not  be  present  in 
thermal  imagery.  For  instance,  Nandhakumar  and  Aggarwal  [1988]  presented  a  technique  to 
segment  thermal  and  visual  images  by  matching  corresponding  features  in  the  two  images. 
Similarly,  Tong  et.  al.  [1988]  developed  a  method  to  integrate  Forward  Looking  Infra-Red  (FLIR) 
sensors  with  Laser  radar  imagery  to  segment  targets  for  target  recognition. 

Multi-sensor  fusion  has  been  often  applied  in  the  recent  past  to  robotics  in  the  context  of  navigation 
and  recognition.  Allen  [1987]  has  developed  a  robotic  system  to  recognize  small  objects  using 
exploratory  tactile  sensing  and  three  dimensional  visual  information.  The  system  is  model-based 
and  components  of  the  models  are  constructively  added  by  combining  tactile  information  and 
segmented  surfaces  of  the  objects.  Earlier,  Crimson  and  Lozano-Perez  [1984]  used  model-based 
vision  to  fuse  information  from  tactile  and  range  sensors  for  measuring  the  position  and  computing 
the  surface  normals  to  identify  and  locate  known  objects.  The  Autonomous  Land  Vehicle  (ALV) 
project  at  Carnegie  Mellon  University  [Goto  and  Stenz  1987]  is  another  example  of  using  multiple 
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sensors  including  stereo  cameras,  laser  range  finders  and  sonar  sensors  for  obstacle  avoidance  and 
outdoor  navigation. 

Neural  networks  also  have  played  a  critical  role  in  sensor  fusion  research.  For  example,  Rajapakse 
and  Acharya  [1990]  proposed  a  hierarchical  network  to  achieve  fusion  of  images  from  multiple 
sensors  for  object  classification.  The  hierarchical  neural  system  based  on  the  Neocognitron  and 
ART  1,  has  many  forward  paths  to  carry  information  from  different  sensors  and  a  fusion  path  to 
carry  the  combined  information.  The  final  decision  is  reached  by  looking  at  the  information  in  the 
fusion  path.  They  have  not,  however,  reported  experimentation  with  any  real  imagery.  Also,  the 
images  from  multiple  sensors  are  assumed  to  have  the  same  characteristics  although  with  different 
scaling,  rotation  and  noise.  This  may  be  an  overly  simplified  representation  of  the  real  world. 


Our  work  in  sensor  fusion  began  in  1990  with  an  SBIR  effort  for  extraction  of  natural  features 
from  SAR  and  visual  imagery  [Raghavan  et.  al.  1991].  We  used  Gabor  filters  [Daugman  1988]  to 
represent  textures  from  co-registered  multi-sensor  imagery  and  neural  networks  to  recognize  the 
terrain  features  from  normalized  textural  feature  vectors.  The  neural  networks  are  trained  to 
classify  the  textural  feature  vectors  from  both  radar  and  optical  images  into  forests,  water,  and 
fields.  We  further  developed  man-made  terrain  feature  extraction  algorithms  using  a  hybrid 
approach  of  traditional  image  understanding  algorithms  and  fuzzy  logic  methods  in  another  Navy 
SBIR  Phase  II  effort  [Raghavan  et.  al.  1996].  This  Phase  1  STTR,  in  contrast  with  our  earlier 
work,  has  been  aimed  at  developing  algorithms  for  SAR  based  cueing  of  passive  imagery  for  target 
recognition. 
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4.  IMAGE  REGISTRATION 


Registration  of  images  is  a  task  of  fundamental  importance  in  the  field  of  image  understanding 
(lU).  The  basic  objective  is  to  be  able  to  relate  the  information  in  one  image  to  the  information  in 
another.  There  are  many  applications  where  the  two  images  are  obtained  using  two  different  types 
of  sensors.  In  remote-sensing  and  image  exploitation  (IE)  applications,  the  sensors  of  interest  are 
Synthetic  Aperture  Radar  (SAR),  Forward-Looking  Infra-Red  (FLIR)  Electro-Optic  (EO),  and 
interferometric  SAR  (BFSAR).  These  sensors  respond  to  scene  characteristics  in  different,  and 
often  complementary,  ways.  It  is  therefore  beneficial  to  use  data  from  more  than  one  type  of 
sensor  in  any  lU  application.  This  type  of  sensor  integration  is  possible  only  if  the  two  images  are 
co-aligned,  or  "registered"  with  respect  to  a  common  coordinate  system.  Of  course,  for  this  to  be 
meaningful,  the  images  should  share  a  significant  amount  of  information  pertaining  to  the  scene 
being  imaged.  Automatic  image  registration  relies  on  detecting  and  exploiting  this  common 
information. 

Brown  [1992]  provides  a  thorough  survey  of  registration  problems,  classifying  them  into  four 
categories;  multimodal,  template,  viewpoint  and  temporal.  This  effort  deals  with  the  specific 
problem  of  multimodal  (i.e.  multisensor)  registration  [Manjunath  and  Fonseca,  1996,  Rignot  et. 
al.  1991],  which  is  the  registration  of  images  of  the  same  scene  acquired  from  different  sensors. 
Registration  methods  are  characterized  in  [Brown  92]  based  on  three  factors:  the  geometric 
transformation  model  used,  the  type  of  image  variations  the  method  can  handle,  and  the  type  of 
computations  performed  in  order  to  register  the  images. 


Geometric  transformations  can  be  characterized  as  2-D  or  3-D,  and  as  global  or  local.  When  a 
global  transformation  model  is  used,  a  single  set  of  parameters  is  computed  for  the  entire  image. 
In  the  case  of  a  large  image,  it  may  be  necessary  to  subdivide  it  into  subregions,  each  with  its  own 
local  set  of  transformation  parameters.  In  this  effort,  we  restrict  ourselves  to  the  determination  of 
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global  2-D  transformations.  The  image  variations  of  interest  to  us  are  geometric  variations— the 
fact  that  any  given  point  in  the  scene  may  appear  at  different  locations  in  the  two  images — and 
radiometric  or  photometric  variations,  which  pertain  to  the  fact  that  this  point  may  have  a  different 
pixel  value  ("intensity")  in  the  two  images. 

Different  computational  approaches  can  be  used  for  determining  the  registration  parameters  once  a 
transformation  model  is  assumed.  Methods  can  be  broadly  classified  as  area-based  or  feature- 
based.  Area-based  methods  assume  that  corresponding  pixels  in  the  two  images  will  have  strongly 
correlated  photometric  values.  A  region  in  one  image  can  thus  be  compared  to  a  region  in  another 
using  some  measure  of  similarity  such  as  cross-correlation.  Feature-based  methods  use 
information  from  discrete  features  such  as  lines,  contours  and  comer  points  extracted  from  the  two 
images. 

Most  traditional  methods  for  image  registration  can  handle  only  minor  geometric  and  photometric 
variations.  In  a  multisensor  context,  however,  the  images  to  be  registered  in  general  may  be  of 
widely  different  types,  obtained  by  disparate  sensors  with  different  resolutions,  noise  levels  and 
imaging  geometries.  The  common  or  "mutual"  information,  which  is  the  basis  of  automatic  image 
registration,  may  manifest  itself  in  a  very  different  way  in  each  image.  This  is  because  different 
sensors  record  different  physical  phenomena  in  the  scene.  For  instance,  an  infra-red  (IR)  sensor 
responds  to  the  temperature  distribution  of  the  scene,  whereas  a  radar  responds  to  material 
properties  such  as  dielectric  constant,  electrical  conductivity  and  surface  roughness.  Area-based 
methods  are  therefore  inappropriate  for  multisensor  registration.  However,  since  the  underlying  3- 
D  scene  giving  rise  to  the  shared  information  is  the  same,  certain  qualitative  statements  can  be  made 
about  the  manner  in  which  information  is  preserved  across  multisensor  data.  Although  the  pixels 
corresponding  to  the  same  scene  region  may  have  different  values  depending  on  the  sensor,  pixel 
similarity  and  pixel  dissimilarity  are  usually  preserved.  In  other  words,  two  scene  points  that  have 
comparable  pixel  values  in  one  t5q)e  of  image  are  likely  to  have  comparable  pixel  values  in  images 
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of  the  scene  obtained  from  other  sensors.  A  region  that  looks  homogeneous  to  one  sensor,  is  likely 
to  look  homogeneous  to  another,  local  textural  variations  apart.  Regions  that  can  be  clearly 
distinguished  from  one  another  in  one  image  are  likely  to  be  distinguishable  from  one  another  in 
other  images,  irrespective  of  the  sensor  used.  Although  this  is  not  true  in  all  cases,  it  is  generally 
valid  for  most  types  of  sensors  and  scenes.  For  instance,  in  S  AR,  EO  and  IR  images  of  an  area 
containing  a  small  lake  surrounded  by  bare  ground,  the  perimeter  of  the  lake  is  likely  to  be  clearly 
visible,  although  the  photometric  appearance  of  the  lake  and  the  surrounding  region  may  be 
sensor-dependent.  Thus  features  derived  from  shapes  of  2-D  image  regions  contain  mutual 
information  that  can  be  used  for  multisensor  registration.  In  this  effort,  we  approximate  region 
boundaries  by  polygonal  segments,  and  use  these  segments  along  with  high  curvature  points  on 
the  boundaries  as  features  for  registration. 

Man-made  objects  in  a  scene  also  give  rise  to  features  that  are  likely  to  be  preserved  in  multisensor 
images.  For  instance,  building  edges  appear  as  intensity  discontinuities  in  SAR,  EO  and  IFSAR 
images,  whereas  edges  of  roads  appear  as  discontinuities  in  SAR  and  EO  images.  Feature-based 
methods  that  exploit  the  information  contained  in  region  boundaries  and  in  man-made  structures  are 
therefore  of  interest  for  multisensor  registration. 

Feature-based  methods  traditionally  rely  on  establishing  feature  correspondence  between  the  two 
images.  Such  correspondence-based  methods  first  employ  feature  matching  techniques  to 
determine  corresponding  feature  pairs  from  the  two  images,  and  then  compute  the  geometric 
transformation  relating  them,  typically  using  a  least-squares  approach.  Their  primary  advantage  is 
that  the  transformation  parameters  can  be  computed  in  a  single  step,  and  are  accurate  if  the  feature 
matching  is  reliable.  The  drawback  is  that  they  need  to  address  the  highly  non-trivial 
correspondence  problem'.  Given  'N'  features  in  each  image,  the  number  of  possible  one-to-one 
feature  mappings  is  'N',  out  of  which  only  one  can  possibly  be  correct.  Some  heuristics  can  be 
employed  to  reduce  the  number  of  potential  correspondences  [Grimson  1981],  but  this  problem 
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still  remains  intractable,  unless  the  two  images  are  already  approximately  registered,  or  the  number 
of  features  is  small.  In  [Li  and  Manjunath  1995],  multisensor  registration  is  accomplished  by 
matching  contours.  This  approach  will  work  if  the  images  contain  a  small  number  of  clean 
contours.  If  the  data  are  noisy,  however,  the  contours  extracted  may  be  split  up  into  smaller 
fragments,  and  contour  matching  becomes  very  difficult. 

In  this  effort,  we  have  proposed  an  approach  to  multisensor  registration  that  eliminates  the  need  for 
feature  matching.  We  first  decompose  the  original  transformation  into  a  sequence  of  elementary 
stages.  At  each  stage,  we  estimate  the  value  of  one  transformation  parameter  by  a  "feature 
consensus"  mechanism  in  which  all  pairs  of  features  are  considered  as  potential  matches,  and  each 
potential  match  votes  for  the  value  of  the  parameter  that  is  consistent  with  it.  We  introduce  the 
concept  of  parameter  observability  to  formalize  this  process. 

4.1  THE  FEATURE  CONSENSUS  APPROACH 

The  basic  principle  of  feature  consensus  is  similar  to  that  used  in  the  Hough  transform.  The  Hough 
transform  maps  the  feature  space  into  the  parameter  space,  by  allowing  each  feature  to  vote  for  a 
subspace  of  the  parameter  space.  Clusters  of  votes  in  the  parameter  space  are  then  used  to  estimate 
parameter  values.  Feature  consensus  is  similar  in  the  sense  that  it  involves  the  determination  of 
vote  clusters  in  parameter  space.  It  is  different  from  the  Hough  transform  in  two  important  aspects, 
(a)  The  basic  voting  unit  is  a  feature  pair  (fi,  f2) ,  where  fi  is  from  image  Ii  and  f2  from  image  I2 
and  (b)  The  parameter  space  is  one-dimensional,  and  each  voting  unit  votes  for  exactly  one  point  in 
this  space.  If  0  is  the  parameter  being  estimated,  the  features  'f  that  vote  should  possess  attributes 

'a'  that  are  related  by 

(4.1) 
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where  0  is  a  bijective  function  that  depends  only  on  the  parameter  being  estimated.  In  other  words, 
0  should  be  observable  with  respect  to  some  feature  and  attribute  classes.  We  assume  that  the 

function  is  of  a  form  that  permits  the  parameter  to  be  written  as 

G=h[a^,a2) 

(4.2) 


The  consensus  function  Cq  is  defined  as 


Ce='ZS(h{a„a2))JieF„ 

u 

(4.3) 


where  6  is  the  discrete  impulse  function.  Feature  consensus  is  simply  the  process  of  determining 
the  value  of  0  that  receives  the  maximum  number  of  votes: 

^max=argmax(C0) 


(4.4) 


In  order  to  estimate  the  transformation  parameters  ai  by  feature  consensus,  we  need  to 
reparametrize  the  transformation  into  a  set  of  stages  such  that  at  each  stage  there  is  a  single 
observable  parameter.  In  mathematical  terms,  we  decompose  the  original  transformation  T  into  a 


sequence  of  transformations 


. ..,(^,)=r;;(r;:'(-r;(/.))) 


(4.5) 


where  (bi  b2  •  •  •  bn)  are  functions  of  the  original  transformation  parameters  (ai  a2  •  •  •  an),  of  the 
form 

In  the  simplest  case,  the  'b'  parameters  are  identical  to  the  'a'  parameters.  At  each  stage  T,  there 
should  be  some  feature  attribute  which  is  transformed  in  a  manner  that  is  dependent  only  on  'bi': 

The  parameters  (bi  b2  •  •  •  bn)  are  estimated  sequentially  by  feature  consensus,  and  at  each  stage  'i' 
the  transformation  'T^bi'  is  applied  the  first  image  'Ii',  leaving  us  then  with  the  task  of  estimating 
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the  remaining  parameters  '(bi+i  bi+2  •  •  ♦  bn)'  between  the  transformed  first  image  and  the  second. 
This  is  performed  until  all  the  'b'  parameters  have  been  estimated.  It  is  straightforward  to  estimate 
the  original  'a'  parameters  of  the  transformation  'T'.  In  most  cases,  this  may  not  be  necessary , 
since  the  first  image  can  be  aligned  with  the  second  simply  by  applying  the  last  stage  of  the 
reparametrized  transformation. 


4.2  AN  IMAGE  TRANSFORMATION  MODEL 


A  number  of  choices  are  available  for  the  global  2-D  transformation  between  two  images,  as 
discussed  in  [Wolberg  1990,  Zheng  1993].  The  model  that  is  chosen  depends  on  a  number  of 
factors,  such  as  prior  knowledge  about  imaging  geometries  [Chellappa  1996],  accuracy  required 
and  computational  cost.  In  general,  the  lowest  order  model  (i.e.  the  simplest  model)  should  be 
used.  Computational  cost  increases,  and  algorithmic  robustness  decreases,  with  increase  in  the 
complexity  of  the  model.  Thus,  although  a  more  complex  model  may  be  able  to  capture  a  wider 
range  of  geometric  variations,  the  practical  problems  associated  with  model  complexity  make  a 
simpler  (and  possibly  less  accurate)  model  more  attractive. 


A  decomposition  of  the  from  (4.5),  satisfying  the  observability  constraint  (4.1)  may  not  be 
obvious  for  any  arbitrary  transformation  'T',  except  in  simple  cases  like  the  similarity 
transformation.  A  similarity  transform,  which  is  characterized  by  four  parameters  (rotation  'P', 
translation  'tx',  'ty'  and  scale  's'),  transforms  a  point  '(x,y)'  to  the  point  '(X,Y)',  according  to: 


rx^ 


=s 


^  cosfi 
^-sin(5 


sinfi 

cosp 


vyj 


ftA 


+ 


(4.6) 

Using  the  notation  'p  =  (x,y)'r',  this  transformation  can  be  expressed  as  a  suite  of  four  stages: 
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np)=T,,(T,^{T.(T,(p]))) 

(4.7) 


where 


r»(p)= 


cosP  sin  (5 
-sinjS  cosP) 


tXp)=sp 


T.SPhP^ 


loj 


tAp)=p+ 


0^ 


In  this  case,  the  parameters  of  the  new  transformation  (P,  tx,  ty,  s)  are  identical  to  the  parameters 
of  the  original  transformation.  The  first  parameter  to  be  estimated  is  'P',  the  angle  of  rotation.  This 
parameter  is  observable  from  the  slopes  of  line  and  edge  features  in  the  images.  If  li'  and  'h'  are 
corresponding  line  features  with  slope  angles  '(J)!'  and  ’<1)2'  respectively,  then 

Ts{p)=sp 


(4.8) 


Thus  'p'  can  be  estimated  by  consensus  of  line  features,  and  'Ii'  can  be  transformed  according  to 

(4.8).  The  scale  's'  can  be  estimated  by  consensus  of  pairs  of  point  features,  with  the  distance 'd' 
between  the  two  points  being  the  candidate  attribute: 


The  translational  shifts  'tx,  ty'  are  observed  using  point  feature  location  as  candidate  attribute: 
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ft  "i 

P2=A+  / 

VyJ 

(4.10) 

Let  us  now  consider  a  more  complex  quasi-affine  model,  with  five  parameters  (rotation  P , 
translation  ’tx’,  'ty'  and  scales  'Sx',  'Sy').  The  transformation  can  be  written  as 

^X^fcosp  sinpVs, 
l^-sinj8  cosjSjl^O 

(4.11) 

Proceeding  in  a  fashion  similar  to  that  used  for  the  previous  case,  this  transformation  can  be 
expressed  as  a  suite  of  five  stages: 


np)=7',,(r,(r,(r.(r,(p))))) 


where 


and 


(4.12) 


In  this  case,  the  rotation  'p'  and  scales  'Sx',  'Sy'  cannot  be  observed  independently  of  each  other 
from  the  slopes  and  lengths  of  segments,  as  in  the  previous  case.  The  solution  to  this  problem  is 
to  find  other  feature/attribute  classes  to  observe  'P'  and  scales  'Sx',  'Sy'  independently,  or  to 
reparametrize  the  transformation  with  respect  to  a  new  set  of  parameters  that  are  all  independently 
observable.  However,  if  the  two  scale  factors  are  approximately  equal  (as  is  true  in  most  cases), 
the  previous  method  for  observing  the  rotation  angle  is  still  valid,  and  the  scale  factors  can  be 
observed  by  a  simple  extension  of  the  previous  method. 
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4.3  NOISE  CONSIDERATIONS 


The  feature  consensus  scheme  may  generate  a  large  amount  of  noise,  particularly  if  the  number  of 
features  is  large.  If  there  are  'N'  features  in  each  image,  there  are  a  total  of  'N^'  votes,  of  which 
only  'N*  can  possibly  be  correct.  Although  this  is  an  improvement  over  the  N!  complexity 
associated  with  correspondence-based  methods,  the  "signal"  component  of  the  consensus 
function  is  small  compared  to  the  "noise"  component.  However,  the  signal  does  not  get  lost  in  the 
noise,  because  the  ’N^-N’  incorrect  votes  will  be  dispersed  over  a  wide  range  of  values,  whereas 
the  'N'  correct  votes  will  (ideally)  be  clustered  around  the  true  parameter  value.  Thus  the  mode  of 
the  distribution  of  votes  can  be  identified.  For  instance,  if  we  are  estimating  rotation  by  comparing 
the  slope  angles  of  line  features,  the  incorrect  votes  will  be  distributed  evenly  in  the  range  \-tU2, 
7t/2',  whereas  the  correct  matches  will  vote  for  the  true  rotation  angle  'p .  Nonetheless,  it  would  be 
helpful  to  reduce  the  number  of  incorrect  votes.  If,  by  some  simple  matching  scheme,  we  can 
restrict  each  feature  to  have  a  maximum  of 'm  «  N’  possible  matches,  we  can  reduce  the  total 
number  of  votes  to  'Nm'.  For  a  detailed  discussion  of  robust  feature-matching  techniques,  see 
[Stewart  1994]. 

4.4  RESULTS  OF  THE  REGISTRATION  ALGORITHM 

Our  system  for  estimation  of  transformation  parameters  is  illustrated  schematically  in  Figure  4.1. 


4.4.1  Feature  extraction 


Robust  extraction  of  features  is  important  for  the  success  of  any  feature-based  multisensor 
registration  scheme.  The  feature  consensus  method  proposed  here  relies  on  the  mutual  information 
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Figure  A.l:  A  schematic  diagram  of  our  feature-based  registration  technique 
(applied  to  Kirkland  SAR— EO  dataset) 
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contained  in  the  extracted  features  in  the  two  images.  In  other  words,  a  large  fraction  of  the 
features  extracted  should  be  common  to  both  images.  This  ensures  that  the  votes  from  matching 
feature  pairs  are  not  lost  in  the  combinatorial  noise  generated  in  the  feature  consensus  process. 
Feature  extraction  has  been  one  of  the  most  widely  researched  areas  of  image  processing,  and  a 
number  of  methods  have  been  proposed.  In  our  current  implementation,  contours  are  extracted 
from  the  images  using  the  Rosin  method  [Venkatesh  and  Rosin  1995].  Some  post-processing  is 
performed  to  eliminate  noisy  contours.  Lines  are  obtained  by  a  polygonal  approximation  of 
contours,  and  then  discarding  small  segments.  Feature  points  are  extracted  from  contours  using 
curvature  as  criterion.  As  an  example,  we  show  the  original  SAR  and  the  EO  images  in  Figures 
4.2a  and  4.2b,  and  the  extracted  features  are  shown  in  Figures  4.3a  and  4.3b. 

4.4.2  Estimation  of  Rotation 


Rotation  is  estimated  by  consensus  of  polygonal  segments,  with  slope  angle  as  the  attribute.  Each 
pair  '(si>  S2)'  of  segments,  'si'  from  'Ii'  and  'S2'  from  T2'*  produces  one  vote  for  the  angle  of 
rotation.  The  vote  is  weighted  by  the  product  of  the  lengths  of  the  segments.  The  consensus 
function  for  this  operation  is  shown  in  Figure  4.4.  Notice  the  sharp  peak  in  the  consensus 
function  near  +100  degrees. 


4.4.3  Estimation  of  Scale  Parameters 

According  to  the  formulation  we  explained  earlier,  two  stages  of  observation  are  required  to 
estimate  the  two  scale  factors  'Sx'  and  'Sy'.  These  two  stages  can  be  combined  into  a  single  stage. 
The  candidate  feature  class  consists  of  pairs  of  feature  points  <(x,y),  (X,Y)>  from  the  same  image. 
The  observation  equations  for  the  scale  factors  are  simple: 
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Figure  4.2a: 


Original 


EO  image  of  the 


Kirkland  dataset 


Figure  4.2b:  Original  SAR  image  of  the  Kirkland  dataset 
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Figure  4.3a:  Features  of  the  EO  image  shown  as  lines  and  comers 


Figure  4.3b:  Features  of  the  SAR  image  shown  as  lines  and  corners 
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Figure  4.4:  Consensus  function  foi 
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The  consensus  function  for  the  scale  parameters  is  shown  in  Figure  4.5. 

The  'x'  and  'y'  scaling  factors  are  approximately  2.1  and  1.6.  Whereas  the  consensus  function  for 
'Sx'  exhibits  an  unambiguous  peak  near  the  true  value,  there  are  two  peaks  in  the  consensus 
function  for  'Sy'.  Currently,  the  false  peak  is  eliminated  by  inspection.  We  need  to  devise  some 
methods  to  eliminate  outliers  to  automatically  eliminate  such  false  peaks. 


4.4.4  Estimation  of  Translation 

Once  the  rotation  and  scale  have  been  estimated,  estimation  of  the  translation  is  straightforward. 
Translational  shifts  are  directly  observed  from  the  positions  of  feature  points  in  the  two  images. 
The  consensus  function  is  shown  in  Figure  4.6.  The  final  output  of  the  registration  process  is 
shown  in  the  SAR  image  (see  Figure  4.7)  with  superimposed  contours  from  the  EO  image.  The 
peaks  in  the  consensus  functions  for  'tx'  and  'ty'  are  unambiguous.  The  result  of  applying  this 
translation  is  shown  by  overlaying  SAR  contours  on  the  EO  image. 

4.4.5  Refinement 

After  carrying  out  the  previous  stages  of  processing,  the  SAR  and  EO  images  are  approximately 
aligned.  This  is  sufficient  for  many  applications.  However,  if  further  accuracy  is  required,  we  can 
perform  a  simple  nearest-neighbor  matching  of  lines  or  feature  points,  and  recompute  the 
transformation  parameters.  Sinee  this  is  a  correspondence-based  technique,  we  can  use  a  more 
complex  transformation  model  than  was  used  for  the  feature  consensus  approach.  In  our  system. 
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Translation  in  X  direction 


Translation  in  Y  direction 
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Translation  in  Y  direction  is  125.6 


Figure  4.6:  Consensus  function  for  the  translation  parameters 


Figure  4. 


:  Final  output  ,  of  the  registration  algorithm, super- imposing 

SAR  features  on  the  EO  image 
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we  use  a  projective  transformation  model  for  this  stage,  in  which  a  point  '(x,y)'  is  transformed  to 
'(X,Y)'  according  to 

(4.13) 


where  'c'  is  an  arbitrary  scalar,  and 

flu  fljl  ^1 
^12  ^22  ^2 
0)3  023  ^ 

For  each  point  correspondence,  we  can  write  two  equations: 

X=a^^x+ a^^y+a^x-axiXX-a^yX 

y  =  +  022y ^32  “  ^233^^ 

(4.14) 

Thus,  we  get  two  equations  for  each  point  correspondence  ’  ((x,y),  (X,Y))  A  minimum  of  four 
such  correspondences  are  therefore  needed  to  solve  for  the  eight  transformation  parameters. 


X,  y,  1  0  0  0  —  x,X,  -yjX, 

0  0  0  ^2  y2  1  -•^2^2  ~yt^i 

(4.15) 

where 

^“(^11  ^21  ^1  *^12  ^22  ^32  ^13  ^23) 

and 

/>=(x,  y,  X,  Y,  X,  K,  X,  r.)' 


Similarly,  given  a  pair  of  matching  lines  1  =  (a  b  c)^'  and  'L  —  (A  B  C)^ ,  with 
[a  b  c)(x  y  l)^  =  0 
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(A  B  C)(X  Y  lf=0 
we  can  write  the  following  two  equations: 


'0  Ac  —Ab  0 
-Ac  0  Aa  —Be 


Be 

0 


—Bb  0  Cc 
Ba  —Cc  0 


^  Cb^ 

y-Caj 


(4.16) 


Four  line  correspondences  are  therefore  required  to  solve  for  the  parameters  of  the  projective 
transformation. 


In  our  implementation,  we  use  a  nearest-neighbor  approach  to  obtain  potential  point 
correspondences,  and  then  estimate  the  transformation  parameters  using  (4.1 1).  Almost  invariably, 
there  will  be  some  false  matches  that  will  lead  to  errors  in  the  estimates.  A  number  of  methods  are 
available  to  detect  and  prune  out  such  "outliers"  [Stewart  1994].  We  use  the  iterative  refinement 
approach  developed  in  [Zheng  1993].  In  this  approach,  transformation  parameters  are  first 
estimated  using  the  available  candidate  point  correspondences.  The  computed  transformation 
parameters  are  used  to  project  points  in  the  first  image  onto  the  second.  A  match  '(pi,  P2)'  is 
considered  to  be  correct  if  the  projection  of  'pi'  does  not  lie  too  far  from  p2 .  Matches  that  fail  to 
satisfy  this  constraint  are  eliminated,  and  the  transformation  parameters  are  recomputed.  This 
estimate-and-prune  step  is  repeated  until  all  matches  satisfy  the  constraint. 

In  essence,  we  have  proposed  an  approach  to  multisensor  registration  that  does  not  rely  on  feature 
correspondence  as  its  primary  mechanism.  We  argue  that  feature  correspondence  cannot  be  reliably 
determined  in  images  obtained  from  disparate  sensors,  and  hence  we  have  proposed  a  method  that 
is  based  on  feature  consensus.  By  considering  all  pairs  of  features  as  potential  matches,  and  by 
allowing  them  to  vote  for  the  transformation  parameters,  we  eliminate  the  need  for  correspondence. 
By  decomposing  the  transformation  into  a  sequence  of  elementary  stages,  we  avoid  the  complexity 
associated  with  Hough  transform-style  methods.  We  introduced  the  notion  of  parameter 
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observability  to  analyze  the  relationship  between  features,  attributes  and  transformation  parameters. 
We  presented  results  on  real  data  to  validate  this  approach.  Further  work  in  needed  on  developing 
better  feature  detectors  for  multisensor  imagery,  and  in  developing  a  comprehensive  taxonomy  of 
features,  attributes  and  transformations. 


5.  CFAR  DETECTION  OF  FOA 

Many  terrain  objects,  particularly,  man-made  objects,  are  fairly  strong  scatterers.  However,  the 
intensity  of  their  radar  returns  heavily  depend  on  imaging  geometry  and  thus  simple  thresholding 
operations  will  not  work.  Constant  False  Alarm  Rate  (CFAR)  processing  of  S  AR  data  is  useful  in 
detecting  the  metallic  objects  and  other  bright  reflectors  such  as  the  buildings  in  spatially  non- 
homogeneous  clutter.  In  CFAR  processing,  the  backscatter  magnitude  at  the  cell  or  the  region 
under  test  is  compared  to  an  adaptive  threshold,  derived  from  a  background  threshold  window. 
There  are  several  versions  of  the  CFAR  that  can  be  used  for  this  purpose  and  we  describe  two  of 
them  that  were  implemented  and  tested.  The  methods  used  are  called  order  statistic  (OS)  CFAR 
and  cell  averaged  (CA)  CFAR. 

The  CFAR  is  essentially  a  binary  hypothesis  problem  that  can  be  described  as  the  presence  or  the 
absence  of  the  object  as: 

HO  :  Target  absent  (Clutter  only ) 

HI:  Target  present. 

The  decision  reduces  to  the  likelihood  test  of  the  form: 

> 

^0  ^ 
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where  Kq  is  the  detector  output  for  the  cell  under  test,  z  is  the  statistic  derived  from  the  detector 
outputs  from  M  cells  in  the  background  reference  window,  and  T  is  the  adaptive  threshold 
multiplier.  Adaptive  threshold  T  is  related  to  a  given  False  Alarm  rate  Pf.J^  as  ; 

oo 

0 

(5.2) 

where  Prg  is  the  probability  under  the  null  hypothesis . 


The  K  distribution  arises  when  a  complex  Gaussian  process  is  modulated  by  a  Chi  distributed 
process.  Equivalently,  it  can  be  shown  to  arise  from  modulating  the  power  of  a  Rayleigh 
magnitude  process  with  a  Gamma  distributed  variable.  This  is  the  case  when  clutter  in  a  given  cell 
exhibits  rapid  Rayleigh  fluctuations,  whose  mean  is  varying  slowly  over  time  according  to  the 
Gamma  distribution.  The  two  parameter  K  pdf  is  given  by: 


Mx)  =  -^^{cxyK,_,{2cx)U(x) 


(5.3) 


where  v  is  the  shape  parameter,  is  the  modified  Bessel  function  of  the  second  kind  of  order 
V  ,  and  c  is  a  power  parameter  related  to  the  mean  clutter  power  Pg  by  Pq  =  ^2  •  The  shape 


parameter  v  controls  the  "spikiness"  of  the  clutter  with  the  lower  value  of  v  implying  more  spiky 
clutter.  For  the  K  distribution,  the  False  Alarm  equation  can  be  written  as: 


FA 


c  2c 
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(TzyK,(2cTz) 


\fz{z)dz 


(5.4) 


29 


For  the  OS  CFAR  processor  based  on  a  single  ranked  sample,  the  test  statistic  is  the  kth  ordered 
statistic  from  the  M  reference  cells.  Hence  the  probability  density  function  (pdf)  of  the  k  th  ordered 
statistic  is  given  by: 


(M 


F,{u)  =  k\  ,  /o(«)Fo(Mr[l-Fo(M)] 


iM-t 


(5.5) 


where  Fo(u)  and  fo(u)  are  the  univariate  clutter  cumulative  density  function  (cdf)  and  pdf  under 
null  hypothesis  Hq.  For  the  K  distribution,  the  kth  order  statistic  is  given  by: 


f,(y)  =  2ck 


■2(cy)"-‘"‘‘ 

^kj 

r(v) 

K,.ficy)K“-‘{2cy) 


k-l 


(5.6) 


which  can  be  simplified  for  v  =  m  + 1  /  2,m  =  0,1,2, . In  particular 


v  =  0.5=>  fy{y)  =  2ck 


exp[-2c(M-k  +  l)y][l-exp(-2cy)]*  * 


(5.7) 


and 


fM\ 


V  =  1.5  =>  friy)  =  4c'fc  ,  exp[-2c(M -k  +  l)y](l  +  2cy)"-*[l - (1  +  2cy)exp(-2cy)J 


(5.8) 


substituting  these  densities  in  the  false  alarm  expression  leads  to  an  expression  for  the  adaptive 
threshold  that  has  to  be  solved  numerically. 


An  example  of  the  CFAR  processing  is  shown  on  a  SAR  image  of  an  urban  area  (see  Figures  5.1 
and  5.2).  The  output  is  binary  in  that  the  areas  of  interest  are  shown  in  white.  Once  the  candidate 
areas  of  interest  are  detected  in  SAR,  the  corresponding  areas  in  visual  or  IR  imagery  are  located 
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Figure  5.2:  Results  of  CFAR  detector  output  shown  as  white  pixels 
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by  applying  the  transformation  computed  using  the  registration  technique  explained  in  Section  4. 
Unfortunately  we  have  not  been  able  to  show  the  entire  process  in  a  single  data  set  due  to  the  lack 
of  co-located  data  set  with  good  resolution. 


6.  MODEL  BASED  RECOGNITION  OF  TARGETS  IN  FOA 

Unrestricted  view  point  object  recognition  has  continued  to  be  a  challenging  problem.  Model- 
based  vision  has  provenly  demonstrated  its  usefulness  in  ATR  applications  by  bringing  the 
complexity  of  object  recognition  down  to  manageable  levels  by  storing  a  priori  knowledge  of  the 
shape  of  the  objects  in  a  database  and  matching  the  model  attributes  to  the  image  variations.  More 
precisely,  the  model-based  approach  involves  an  object-centered  representation  of  three 
dimensional  models  of  objects,  and  a  matching  scheme  to  rotate  and  translate  the  model  to  align  the 
model  with  the  image.  Popular  examples  of  this  approach  include  ACRONYM  [Brooks  1981]  and 
SCERPO  [Lowe  1987]  (see  [Chin  and  Dyer  1986]  for  a  survey). 


An  alternative  approach  to  object  recognition  involves  storing  normalized  multiple  views  of  the 
object  in  the  model  database  and  finding  the  "closest"  viewpoint  appearance  to  the  image  instance. 
This  approach  is  becoming  increasingly  popular  since  it  mitigates  the  correspondence  and  model 
representation  issues.  For  example,  the  recent  work  of  Edelman  and  Weinshall  [1989]  appears  to 
show  that  human  subjects  exhibit  better  performance  in  recognizing  the  objects  when  the  familiarity 
of  multiple  views  of  objects  increases.  We  argue  that  improved  familiarity  does  not  necessarily 
imply  multiple  view  model  storage  since  familiarity  can  also  aid  in  generating  correspondence 
hypotheses  which  in  turn  result  in  constrained  search  of  the  candidate  models.  The  early 
perceptual  grouping  theory  of  Lowe  [1987]  is  an  example  of  how  to  achieve  constrained  search. 
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Irrespective  of  convincing  biological  evidence  for  either  approach,  an  important  computational 
issue  related  to  multiple-view  approach  is:  how  do  we  a  priori  generate  the  critical  viewpoints  or 
'aspects'  to  be  stored?  Or  what  constitutes  a  'critical  view'?  Even  though  the  recent  work  of  Seibert 
and  Waxman  [1992]  discusses  this  issue  and  presents  a  possible  solution  to  this  problem  via 
viewpoint  clustering,  it  is  inadequate  for  a  real-world  problem  of  arbitrary  viewing  positions,  and 
missing  or  additional  features  other  than  object  silhouette  points  found  in  the  image.  On  the  other 
hand,  this  problem  is  nonexistent  in  model-based  vision  resulting  in  a  major  advantage  of  reduced 
memory  requirements.  But,  the  price  we  have  to  pay  for  this  advantage  is  the  additional  need  to 
obtain  the  model-image  transformation  parameters  for  matching  the  candidate  models  and  the 
corresponding  image  points.  This  problem  is  nontrivial  especially  when  the  candidate  list  is 
sufficiently  large,  resulting  in  a  combinatorial  explosion. 

Our  approach  to  solving  this  problem  involves  an  efficient  multi-level  search  method.  As  part  of 
this  multi-level  search  method,  we  have  developed  a  fast  technique  to  compute  the  model-image 
transformation  parameters  given  any  three  or  more  corresponding  model-image  pairs.  The 
motivation  of  our  method  comes  from  the  recent  work  of  Huttenlocher  and  Ullman  [1990]  who 
showed  that  one  can  obtain  the  parameters  for  a  unique  (up  to  reflection)  model-image 
transformation  under  certain  restrictions.  However,  these  restrietions  are  severe  enough  to  limit 
their  wide  usage.  We  have  developed  two  alternatives  for  their  technique,  one  of  which  is  based 
on  matching  feature  points,  and  the  other  is  based  matching  higher  level  features,  namely,  line 
segments.  Before  we  get  into  the  relevant  details,  we  first  would  like  to  provide  an  overview  of 
our  approach. 

A  schematic  diagram  of  the  model-based  recognition  of  targets  is  shown  in  Figure  6.1. 
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Image  frame 


Figure  6.1:  The  model-based  approach  to  recognition  of  objects  within  FOA 

The  recognition  process  involves  selection  of  models  (indexing)  based  on  geometric  invariance 
followed  by  the  alignment  of  models  with  the  image.  Assuming  the  resolution  of  the  image 
sequence  is  fairly  sufficient,  we  plan  to  use  the  BRL  CAD  models  for  aligning  the  models  with  the 
image.  We  have  developed  two  different  techniques  to  do  this  alignment,  one  of  which  is  point 
based  and  the  other  is  line  based.  We  begin  with  the  point  based  formulation. 


6.1  POINT-BASED  FORMULATION 

The  image  projection  is  assumed  to  be  the  scaled-orthographic.  That  is,  we  approximate  the 
perspective  projection  by  the  orthographic  projection  with  a  scaling  factor.  Thus  we  have. 
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(6.1) 


where  (X,  Y)  is  a  point  in  the  image  which  corresponds  to  the  point  (x,  y,  z)  in  the  model  and  s  is 
the  scale  factor.  This  approximation  is  valid  for  most  real-world  applications  especially  in  the  case 
of  air-ground  target  acquisition  where  Az/z  is  expected  to  be  small  when  compared  to  z.  We  follow 
the  convention  that  when  we  specify  a  quantity  in  the  object-centered  frame  of  reference  we 
subscript  the  quantity  with  'o'.  For  a  point  (Xq  Zq)  in  the  object-centered  frame  of  reference, 

the  corresponding  point  projected  in  the  image  plane  according  to  the  transformation,  referred  to  as 
the  jc-transformation,  is  given  by: 


’X' 

fr  1 

'xo' 

Y 

=  s 

T 

y 

+K 

yo 

. 

(6.2) 

where,  contains  the  first  two  rows  of  the  rotation  matrix  R.  Typically  R,  in  the  roll-yaw-pitch 
formulation  with  Py,  p^)  as  the  rotation  angles.  It  is  clear  from  (4)  that  the  ;c-transformation 
involves  six  parameters  (s,  T^,  Ty,  p^,  liy,  Pz).  Intuitively,  it  follows  that  to  solve  for  the  six 
parameters,  we  require  at  least  three  corresponding  model-image  pairs  which  generate  six  error 
quantities  between  projected  image  coordinates  of  the  model  points  and  actual  image  points. 


Our  algorithm  to  compute  the  model-image  transformation  is  a  Newton's  method  in  that  it  involves 
computing  a  set  of  error  quantities  which  are  then  used  to  drive  a  nonlinear  system  towards  its 

solution  in  the  direction  of  minimizing  these  error  quantities.  More  precisely,  if  a  solution  for  the 
nonlinear  system  of  variables  'U'  (i.e.  U  =  [s,  T^,  Ty,  Hy  ,  T)  is  required,  Newton's 

method  allows  the  following  iterative  error  correction  such  that 

ui+1  =U^-c, 


(6.3) 
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where  the  required  correction  c  is  related  to  the  model-image  error  'e'  through  the  Jacobian  'J'  of 
partial  derivatives  (3e^/9cj)  as  follows.  That  is, 

Jc  =  e. 

(6.4) 


The  error  vector  used  to  correct  the  current  estimates  is  simply; 


(6.5) 

The  correction  vector  c  can  then  be  obtained  by  solving  the  linear  system  in  (6.4).  Note  that  Rc 
contains  only  the  first  two  rows  of  the  rotation  matrix  R  and  thus,  it  is  a  2  X  3  matrix.  The 
Jacobian  'J'  contains  two  columns  corresponding  to  only  one  of  the  three  pairs  of  the  model-image 
tuples.  It  may  be  noted  that  to  solve  the  linear  system  of  six  unknowns  in  the  correction  vector  'c' 
we  need  to  obtain  the  complete  Jacobian  by  adding  the  columns  due  to  the  remaining  two 
corresponding  pairs.  A  major  advantage  of  the  proposed  iterative  technique  is  its  fast  convergence. 
Now,  in  order  to  extend  this  approach  to  a  least-squares  formulation,  we  only  need  to  add  two 
additional  columns  to  j”^  and  two  additional  rows  to  'e'  for  every  additional  model-image  pair. 
However,  since  there  are  only  six  unknowns  the  system  becomes  over-constrained.  By  pre¬ 
multiplying  (6.4)  by  j"*'  we  get; 

jTjc  =  jTe 

(6.6) 

We  have  implemented  this  algorithm  for  a  sequential  machine  and  have  found  that  the  convergence 
occurs  rapidly  within  a  few  iterations. 
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6.2  LINE-BASED  CLOSED-FORM  FORMULATION 


The  point  based  formulation  has  a  major  drawback  reliable  extraction  of  feature  points.  Often, 
feature  points  are  difficult  to  extract  from  noisy  images.  In  such  cases,  lines  are  easier  to  extract  as 
long  as  we  do  not  attach  any  significance  to  the  end-points  of  these  lines.  That  is,  we  treat  each 
line  segment  as  though  it  is  an  infinite  line  with  a  certain  intercept  and  slope.  This  way,  the  line 
based  formulation  is  fairly  robust. 


X' 

Let  P,.  = 

and  Pj  = 

2.. 

X. 

be  two  end  points  in  3-D  which  define  the  line  segment  of  interest  to 


us.  The  projections  of  these  points  are  related  to  their  3-D  counterparts  as: 


Pi 
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T 
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(6.7) 


For  each  model  line  Lk  with  its  end  points  in  3-space  and  corresponding  end  points 

( Pkt » the  2-d  image  we  can  show  that 


=s 


-PkS' 
^(Py^Pk,  ~Pk.,)) 


(6.8) 


After  some  algebra  involving  this  equation  for  the  two  end  points  of  the  three  lines,  we  can  show 
that: 

AR.=BRy 


(6.9) 


where  the  3X3  matrices  A  and  B  are  given  by: 


38 


'(yl-yl)(Pl-Plf ' 

Uj-4)(P,'-Pl)"  ■ 

A  = 

(y^-yDiP^-PiY 

andB  = 

iyl-ylWl-Plf . 

_{x\-xlWl-Plf  _ 

Assuming  B  is  inveitable,  we  can  introduce  'W  such  that  W  =  B'^A.  Now,  hy  imposing  the 
orthogonality  and  the  unit-length  constraints,  we  get; 


R^R^=RlW^WR,=l 

r:w%=o 


(6.10) 


(6.11) 

It  may  he  noted  that  W  contains  all  the  known  quantities  relevant  to  the  three  model  line  image  pairs 
in  consideration.  The  solution  to  (6.10)  and  (6.1 1)  can  he  reduced  to  a  solution  for  a  fourth  order 
polynomial.  To  demonstrate  this  we  first  perform  an  SVD  on  W, 


W=UAV={u^ 


«2  M3) 
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V 
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0  0 
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(Vi  V2 


Since  V  is  the  SVD  is  also  an  orthonormal  basis  of  R3,  we  can  write  Rx  =  avi  +  bv2  +  CV3.  With 

we  get  the  system: 

a^+b^+c^=l 
XyC^  +  ^2^*^  +  X’^C?  —  1 

XiWj^a^+X2W22b^+X^w^^c^+{X^W2^+X2W^2)ab+{X2W^2+^3^2i)bc+{X2Wi^+X^w^i)ac=0 

(6.12) 
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With  some  tedious  algebra,  we  can  show  that  the  rotation  parameters  involving  the  rotation  matrix 


R  reduce  to  a  4th  order  polynomial: 


(6.13) 


After  we  solve  for  the  rotation  parameters  from  the  above  equation,  we  can  back-substitute  in  the 
projection  equation  to  get  the  scale  and  translation  parameters.  We  have  avoided  some  tedious 
math  and  substitutions  for  clarity. 


The  above  quartic  equation  will  have  zero,  two  or  four  non-negative  roots  that  can  give  rise  to 
between  zero  and  32  real-valued  solutions  to  the  system  (6.12  with  back  substitutions)  in  the 
general  case.  The  true  solution  will  always  emerge  in  the  zero  noise,  correct  correspondence  case. 
For  each  non-negative  solution  a  to  the  quartic,  there  is  upto  one  non-negative  value  of  P  and  upto 
one  non-negative  value  of  c^  that  can  be  computed  from  the  first  two  equations  of  the  system 
(6.12).  Since  each  valid  set  (a^,  b^,  c^)  gives  rise  to  8  possible  alternatives  (±a,  ±b,  ±c),  and  there 
are  upto  four  such  sets,  there  are  potentially  32  solutions.  Validity  of  each  of  the  solutions  is 
verified  by  evaluating  the  third  equation  of  (6.12).  Once  the  values  of  (a,b,c)  are  known,  the 
rotation  vectors  Rx,  and  Ry  can  be  determined.  The  scale  factor  's'  can  be  computed  from  (6.8). 
Also  the  translations  Tx  and  Ty  can  be  recovered  from  (6.7)  after  substituting  for  the  scale  and  the 
rotation  quantities. 


6.3  SOLVING  THE  CORRESPONDENCE  PROBLEM 

It  may  be  noted  that  both  the  line  based  and  the  point-based  alignment  methods  help  solve  the 
problem  of  pose  estimation  give  the  sets  of  corresponding  model-image  feature  pairs.  We  still 
have  not  solved  the  model-image  correspondence  problem.  The  model-image  correspondence 
problem  is  a  very  challenging  and  complex  problem  by  itself,  requiring  a  lot  more  attention  than  a 
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Phase  I  effort  can  focus.  However,  we  have  investigated  the  underlying  complexity  and  developed 
an  interim  solution  which  works  well  for  a  limited  model  set. 


We  begin  explaining  this  method  by  making  an  important  observation  as  follows.  When  there  are 


N  model  and  M(<N)  image  lines,  there  are 


Ni  '' 

(W-I)!. 


permutations  of  model  to  image  lines. 


However,  not  all  these  permutations  lead  to  the  correct  solution(s).  There  are  several  simple 
heuristics  that  we  can  employ  to  solve  this  combinatorial  problem.  For  instance,  on  the  basis  of 
adjacency,  the  permutation  can  be  reduced  to  a  much  smaller  quantity.  This  is  performed  by 
mapping  triplets  of  adjacent  image  lines  with  triplets  of  model  lines  having  a  common  vertex.  The 
search  space  can  be  substantially  reduced  this  way.  We  can  also  compare  the  ratios  of  lengths  of 
image  lines  meeting  at  an  intersection  with  those  of  the  hypothesized  model  pairs  and  prune  out  the 
impossible  pairs.  There  are  several  other  such  heuristics  which  can  be  used  to  reduce  the  search 
space. 


Once  the  search  space  is  reduced,  we  initiate  a  following  clustering  technique  (similar  to  a  Hough 
transform)  on  the  constrained  search  space  involving  all  the  normalized  rotation  angles  (rx,  ry,  r^) 
as  follows. 

1  Initially,  the  entire  "cloud"  of  points  is  considered  to  lie  on  a  single  cluster.  The 
size  of  the  cluster  is  set  to  1.0  (since  the  normalized  rotation  angles  lie  within  the 
unit  cube. 

2  At  each  iteration,  the  center  of  mass  of  the  current  cluster  is  computed.  The  cluster 
size  is  shrunk  by  a  constant  factor  (usually  a  number  close  to  1.0,  say  0.95). 


41 


3  The  iterations  are  repeated  till  the  cluster  size  shrinks  to  a  small,  predetermined 
value  (say  0.05).  The  centroid  of  the  resulting  cluster  is  chosen  to  be  the  cluster 
center. 

Using  the  above  combinatorial  process  for  guessing  matching  triplets,  followed  by  a  closed  form 
solution  to  the  guess  and  clustering  on  all  solutions  obtained  gives  the  cluster  center.  Often,  we 
have  experimentally  observed  to  correspond  to  the  true  solution.  To  introduce  additional  noise 
robustness,  we  can  then  use  these  initial  guess  and  all  the  high  confidence  model-image  feature 
pairs  in  a  non-linear  least-squares  method. 

In  the  Phase  11  effort  for  final  prototype  development,  we  will  replace  strengthen  this  approach 
with  additional  heuristics.  Also,  the  model  selection  process  will  be  improved  by  using  geometric 
invariants  [Weinshall  1993,  Weiss  1993]  and  simple  cues  such  as  aspect  ratios  of  targets  for 
heuristics  based  ranking.  We  will  also  investigate  other  major  issues  such  as  the  alignment  metric. 

6.4  RESULTS  OF  MODEL-BASED  ATR 

We  show  the  results  of  our  model-based  ATR  module  on  the  MUSTRS  data  set.  The  original 
FLIR  image  of  the  zil  truck  with  the  lines  extracted  is  shown  on  Figure  6.2.  The  clusters  of  closed 
form  solutions  visualized  from  different  perspectives  are  shown  in  Figure  6.3a  and  6.3b.  The 
"bunching"  of  the  solutions  can  be  clearly  seen  at  the  top  of  the  image.  Using  the  cluster  center, 
the  rotation  angles,  scale  and  translation  were  obtained  and  these  were  passed  to  the  non-linear 
least-squares  module.  The  recognized  3-D  model  of  the  zil-truck  at  the  end  of  the  model-based 
alignment  process  shown  in  Figure  6.4.  The  mirror  as  well  as  true  solutions  are  shown  in  Figure 
6.5.  The  mirror  solution  can  be  removed  only  if  we  consider  more  sophisticated  alignment 
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Figure  6.2:  Lines  extracted  on  a  FLIR  image  of  the  MUSTRS  data  set 
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Figure  6.3b:  Visualization  of  the  clusters  in  the  parameter  space 
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Figure  6. A:  Results  of  aligning  the  wire-frame  model  on  the  IR  image 
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Figure  6,5:  The  correct  solution  and  its  mirror  reversal  aligned 
on  the  IR  image  of  the  zil~truck 


metrics.  With  the  Euclidean  measure,  both  the  solutions  will  have  strong  peaking  in  the  parameter 
space. 


7  REAL-TIME  FEASIBILITY 


To  demonstrate  the  real-time  feasibility  of  the  algorithms  developed  in  this  STTR,  we  have 
implemented  the  CFAR  techniques  on  an  embedded  board.  Since  the  computations  are  locally 
parallel  for  these  algorithms,  we  can  make  use  of  a  simple  Single  Instruction  Multiple  Data  (SIMD) 
type  hardware  to  compute  most  operations  in  parallel.  One  such  hardware  is  the  CNAPS  which  is 
quite  inexpensive  and  commercially  available  as  PCI  and  ISA  boards  for  about  the  price  of  a  PC. 

The  CNAPS  system  is  a  parallel  hardware  that  can  be  configured  as  a  dedicated  Ethernet  resource 
for  Sunsparcstations  and  Hewlett  Packard  workstations  with  an  architecture  optimized  for 
executing  image  processing,  pattern  recognition  and  neural  network  algorithms.  Figure  7.1 
illustrates  the  CNAPS  architecture.  This  system  is  a  linear,  single-instruction-multiple-data 
(SIMD)  array  of  state  of  the  art  custom  CMOS  processors  each  equipped  with  a  block  of  local 
memory. 

A  CNAPS  sequencer  supervises  the  execution  of  instructions  on  these  processors.  The  sequencer 
broadcasts  parallel  input  data  to  every  processor  in  the  array  on  an  input  bus,  and  broadcasts 
instructions  on  a  command  bus.  The  processors  execute  the  instructions  simultaneously  on  the 
data  stored  in  their  local  memories. 

The  program  and  file  memory  areas  store  the  assembled  machine  code  and  segments  of  data 
respectively.  A  32-bit  internal  bus  forms  the  interconnection  network  between  the  sequencer, 
program  and  file  memory  areas  and  the  external  interface. 
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The  currently  available  CNAPS  processors  have  a  clock  speed  of  20  MHz.  The  largest  of  these 
configurations  has  512  processors,  giving  a  throughput  of  approximately  10  billion  instructions 
per  second.  The  complex  multiply-accumulate  instruction  desired  for  neural  network  operations 
can  be  performed  in  one  clock  cycle  as  the  internal  circuitry  on  the  sequencer  splits  the  clock  into 
four  phases.  The  local  memory  on  each  processor  is  currently  4096  byte,  and  the  global  data 
memory  (file  memory)  is  32  Mbyte.  In  addition,  the  Motorola  68030  based  controller  that 
interfaces  between  the  host  and  the  server  provides  4  Mbyte  of  DRAM  for  storing  control 
software. 


Figure  7.1:  The  CNAPS  architecture  for  a  512  processing  node  (PN)  array 

We  have  implemented  the  CFAR  algorithm  on  the  CNAPS  using  the  CNAPS-C  language,  a 
parallel-C  environment.  Though,  neither  optimized  nor  implemented  at  assembly  language  level, 
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this  implementation  still  achieves  a  speed-up  of  roughly  10-26  times  over  a  sequential-C 
implementation  on  the  sparcstation  for  an  image  of  512X512  in  size.  The  performance  statistics 
are  shown  in  Table  7.1.  The  speed-up  we  have  quoted  here  includes  I/O  considerations.  In  the 
case  of  CNAPS,  there  is  a  feature  known  as  Quick  I/O  which  enables  us  to  send  the  data  directly  to 
the  processor  chips  if  we  can  use  the  appropriate  mezzanine  boards. 


Test  image  size 

256  X  256 

#  of  CNAPS  processors  used 

256  (one  row  per  processor) 

Clock  cycles  used  by  the  CNAPS  impl. 

CVCFAR:  690k  cycles  -  35msec 

OSCFAR:  1410k  cycles  -  71  ms 

Clock  cycles  used  by  the  serial  impl.  on 

Sparc  10 

CVCFAR:  690k  cycles  -  86 1  msec 

OSCFAR:  1410k  cycles  -  774  msec 

Table  7.1  -  Performance  Statistics  of  the  CNAPS  vs.  Serial  Implementation  of  the 

CFAR  Algorithms 


8.  PHASE  II  OUTLINE 

This  Phase  I  has  demonstrated  the  concept  that  wide-angle  SAR  can  be  effectively  used  to  cue 
passive  imagery  for  model-based  recognition  of  targets.  Our  experience  with  the  proof-of-concept 
and  the  insights  we  have  gained  during  Phase  I  suggest  that  a  fully  developed  Phase  11  prototype 
will  advance  the  state  of  the  art  to  reach  the  required  level  of  progress  in  ATR  technology  for  target 
acquisition  applications.  Even  though,  object  recognition  from  FLIR  imagery  has  been  studied  for 
well  over  a  decade  now,  most  of  these  efforts  are  less  applicable  to  the  air-ground  target  acquisition 
problem  due  to  the  large  variation  in  target  appearance  from  different  view-angles.  Meanwhile, 
very  interesting  results  have  been  achieved  by  the  model-based  vision  community.  The  notion  of 
motion  as  an  important  cue  is  just  beginning  to  gain  momentum  in  the  community  even  though  we 
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proposed  to  use  motion  for  ATR  in  1989  and  earlier  [Raghavan  et.  al.  1991].  This  Phase  I  work 
has  reinforced  some  of  these  concepts. 


The  primary  technical  objective  of  the  two-year  Phase  n  work  is  to  expand  this  concept  of  motion 
based  ATR  towards  developing  a  prototype.  The  first  year  effort  addresses  several  design  issues 
including  the  multi-resolution  based  optical  flow  computing,  perceptual  grouping  algorithms  for 
hypothesization  of  model-image  corresponding  points,  model-selection  and  indexing  algorithms. 
Some  of  the  tasks  involved  in  the  first  year  of  the  Phase  n  are  outlined  below. 

•  We  would  first  undertake  the  completion  of  the  multi-resolution  based  optical  flow 
computation  for  eliminating  the  dependency  of  desired  frame  rate  of  the  camera  on  the 
expected  range  of  speed  for  targets  in  the  image. 

•  We  need  to  implement  the  perceptual  grouping  theory  algorithms  (step  2  shown  in  Figure  6) 
for  establishing  the  correspondence  between  model-image  points. 

•  The  model-selection  and  indexing  algorithms  need  to  be  (step  1  shown  in  Figure  6) 
implemented  to  enable  automated  selection  of  a  model  for  computing  the  model-image 
transformation. 

•  A  better  error  metric  based  on  feature  information  (e.g.,  local  orientation  of  edges)  need  to  be 
developed  to  match  the  alignment  error  between  the  model  and  image  points. 

The  focus  of  the  second  year  effort  is  to  develop  a  near-real-time  prototype  of  the  model-based 
ATR  system  by  completing  all  the  required  peripheral  modules  designed  in  the  first  year.  The 
following  are  some  of  the  tasks  involved  in  this  stage. 
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•  An  end-to-end  simulation  of  the  prototype  software  system  to  emulate  the  model-based  ATR 
system  will  be  undertaken.  This  ineludes  the  placeholders  for  all  the  modules  of  the  front- 
end  motion  information  processor  and  the  rear-end  model-based  vision  component. 
Primarily,  this  simulation  will  take  place  on  a  serial  machine  like  Sun  Sparcstation. 

•  Parallelizable  modules  of  the  system  (e.g.,  optical  flow  algorithm  if  we  deal  with  video  data) 
will  be  implemented  on  a  parallel  hardware  platform.  LNK  currently  has  access  to  the 
CNAPS  machine  with  512  processors  from  Adaptive  Solutions  Inc.  for  real-time  simulation 
of  parallelizable  algorithms.  This  machine  has  shown  the  potential  for  speed-up  of  the 
sequential  simulations  up  to  a  factor  of  100  or  more  in  our  current  implementations  related  to 
image  classification. 

•  A  full  scale  demonstration  of  the  final  system  will  be  accomplished  with  FLIR  data  acquired 
for  ATR.  This  data  will  be  exclusively  used  for  the  purpose  of  demonstration  and  will  not  be 
used  for  calibrating  the  system. 

In  addition  to  these  major  tasks  required  to  complete  the  prototype,  an  important  task  of  the  Phase 
n  work  is  to  investigate  the  technology  transfer.  Some  of  the  concerns  in  this  regard  include  how 
the  prototype  system  can  be  implemented  on  a  stand-alone  hardware  which  can  be  mounted  aboard 
the  aircraft,  communication  requirements,  and  other  performance  issues. 
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